Run-Time Dynamic Resource Adjustment for Mitigating Skew in MapReduce
نویسندگان
چکیده
منابع مشابه
SkewTune in Action: Mitigating Skew in MapReduce Applications
We demonstrate SkewTune, a system that automatically mitigates skew in user-defined MapReduce programs and is a drop-in replacement for Hadoop. The demonstration has two parts. First, we demonstrate how SkewTune mitigates skew in real MapReduce applications at runtime by running a real application in a public cloud. Second, through an interactive graphical interface, we demonstrate the details ...
متن کاملRun-time mapping: dynamic resource allocation in embedded systems
Many desired features of computing platforms, such as increased fault tolerance, variable quality of service, and improved energy efficiency, can be achieved by postponing resource management decisions from design-time to run-time. While multiprocessing has been widespread in embedded systems for quite some time, allocation of (shared) resources is typically done at design-time to meet the cons...
متن کاملHandling Data Skew in MapReduce
MapReduce systems have become popular for processing large data sets and are increasingly being used in e-science applications. In contrast to simple application scenarios like word count, e-science applications involve complex computations which pose new challenges to MapReduce systems. In particular, (a) the runtime complexity of the reducer task is typically high, and (b) scientific data is ...
متن کاملROUTE: run-time robust reducer workload estimation for MapReduce
MapReduce has become a popular model for large-scale data processing in recent years. Many works on MapReduce scheduling (e.g., load balancing and deadline-aware scheduling) have emphasized the importance of predicting workload received by individual reducers. However, because the input characteristics and user-specified map function of a given job are unknown to the MapReduce framework before ...
متن کاملHandling partitioning skew in MapReduce using LEEN
MapReduce is emerging as a prominent tool for big data processing. Locality is a key feature in MapReduce that is extensively leveraged in dataintensive cloud system: it avoids network saturation when processing large amount of data by co-allocating computation and data storage — the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presen...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Computer Modeling in Engineering & Sciences
سال: 2021
ISSN: 1526-1506
DOI: 10.32604/cmes.2021.013244